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Abstract 

A generalized triangle group is a group that can be presented in the form G = 
{ x,y I = y'^ = w{x, yY = 1 ) where p,q,r > 2 and w{x, y) is a cyclically re- 
duced word of length at least 2 in the free product "Lp-^lq = {x^y \ — y^ — 1) . 
Rosenberger has conjectured that every generalized triangle group G satisfies the 
Tits alternative. It is known that the conjecture holds except possibly when the 
triple is one of (2, 3, 2), (2,4,2), (2,5,2), (3,3,2), (3, 4, 2), or (3, 5, 2). In 

this paper we show that the Tits alternative holds in the case (p, g, r) = (3, 4, 2). 

1 Introduction 

A generalized triangle group is a group that can be presented in the form 

G = {x,y\ = y'' = w{x, = I ) 

where p,q,r > 2 and w{x, y) is a cyclically reduced word of length at least 2 in the 
free product "Lp * "Lq = { x,y \ = y'' = 1 ) that is not a proper power. It was 
conjectured by Rosenberger that every generalized triangle group G satisfies the 
Tits alternative. That is, G either contains a non-abelian free subgroup or has a 
soluble subgroup of finite index. 

It is now known that the Tits alternative holds for a generalized triangle group G 
except possibly when the triple {p,q,r) is one of (2,3,2), (2,4,2), (2,5,2), (3,3,2), 
(3,4,2), or (3,5,2). (See [2] for the case 1/p + l/q + 1/r < 1; see for the case 
r > 3; see ^ for the cases (3,6,2) and (4,4,2); and see P,[H],|5 505^21 for the cases 
(2, g, 2) where q > 6. Many of these results are described in the survey article [H].) 
In this paper we show that the Tits alternative holds in the case (3, 4, 2): 

Main Theorem. Let r = {x,y\x^ = y^ = w{x, y)^ = 1 ) where 'w{x, y) = x'^^y^^ . . . 
x^kyf^k^ 1 < ai < 2, 1 < Pi < 3 for each 1 < i < k where k > 1. Then the Tits 
alternative holds for T. 

Benyash-Krivets and Barkovich i5_ have proved this result when k is even, and 
for this reason we focus on the case when k is odd. 
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2 Preliminaries 



We first recall some definitions and well-known facts concerning generalized triangle 
groups; further details are available in (for example) [H,- 

Let G = ( x,y I = y"^ = w{x,y)'^ = 1 ) where w{x,y) = x"^y^^ . . . x'^'^y^'^ , 
^ ^ Oii < i, 1 < Pi < m for each 1 < i < k where A; > 1. A homomorphism 
p : G ^ H (for some group H) is said to be essential if p{x) , p{y) , p{w) are of orders 
£,ni,2 respectively. By |2] G admits an essential representation into PSL{2,C). 

A projective matrix A G PSL(2, C) is of order n if and only if tr(^) = 2 cos(g7r/n) 
for some {q,n) = 1. Note that in PSL{2,C) traces are only defined up to sign. A 
subgroup of PSL{2, C) is said to be elementary if it has a soluble subgroup of finite 
index, and is said to be non- elementary otherwise. 

Let p : {x,y \ x^ = = 1 ) ^ PSL{2, C) be given hy x X, y where X, Y 
have orders m, respectively. Then w{x,y) i— > w{X,Y). By Horowitz tTw{X,Y) 
is a polynomial with integer coefficients in trX, try, trXy, of degree k in trXy. 
Since X, Y have orders i, m, respectively, we may assume (by composing p with an 
automorphism oi { x,y \ x^ = = 1 ), if necessary), that ixX = 2cos(7r/£), try = 
2 cos(7r/m). Moreover (again by 10 ) X and Y can be any elements of PSL{2, C) with 
these traces. We refer to tr'w{X, Y) as the trace polynomial of G. The representation 
p induces an essential representation G PSL{2, C) if and only if t'!:p{w) = 0; that 
is, if and only if trXy is a root of tTw{X,Y). By ^01 the leading coefficient of 
tru;(X, y) is given by 



Now if X,Y generate a non-elementary subgroup of PSL{2,C) then p{G) (and 
hence G) contains a non-abelian free subgroup. Thus in proving that G contains a 
non-abelian free subgroup we may assume that X, Y generate an elementary subgroup 
of PSL{2,C). By Corollary 2.4 of JH] there are then three possibilities: (i) X,Y 
generate a finite subgroup of PSL{2,C)] (ii) tr[X,y] = 2; or (iii) trXy = 0. The 
finite subgroups of PSL{2,C) are the alternating groups A4 and A^, the symmetric 
group 54, cyclic and dihedral groups (see for example [HI). The Fricke identity 



implies that (ii) is equivalent to trXy = 2cos(7r/^ it vr/m). These values occur as 
roots of trw{X, Y) if and only if G admits an essential cyclic representation. Such a 
representation can be realized as x A,y ^ B where 



sin(aj7r/£) sin(/?j7r/m) 



(1) 



sin(7r/^) sin(7r/m) 



tr[X,y] = {trXf + {trYf + {trXYf - (trX)(try)(trXy) - 2 




We summarize the above as 
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Lemma 1 Let G = {x,y\x^ = = ■w{x,y)'^ = 1 )■ Suppose G — > PSL{2,C) is 
an essential representation given by x X,y Y , where trX = 2cos(7r/£), try = 
2cos(7r/m). If G does not contain a non-abelian free subgroup then one of the follow- 
ing occurs: 

1. X,Y generate ^4,^4, ^5 or a finite dihedral group; 

2. trXy = 2cos(7r/^ ± vr/m); 

3. trXy = 0. 

Case (2) occurs if and only if G admits an essential cyclic representation. 

3 Proof of Main Theorem 

Throughout this section T will be the group defined in the Main Theorem. 

Lemma 2 If T admits an essential cyclic representation then T contains a non- 
abelian free subgroup. 

Proof 

Let p : r ^ Z12 be an essential representation. Then K = kerp has a deficiency zero 
presentation with generators 

ai = yxy'^x'^, 02 = y^xy"^x"\ 03 = y^xy^'^x'^, 
04 = xyxy^^x^'^ , 05 = xy'^ xy~^ x~'^ , uq = xy^xy~^x~'^ , 

and with relators 

W'{ai, . . . , 06, ai, . . . , ai-i)W'{y'^aiy'^, y'^a^y'^, y^aiy^, . . . , y^ai_iy^) (1 < i < 6) 

where W' is a rewrite of W. 

Let 5* = { [aj,aj], aiiy^aiy"^) (1 < i,j < 6)}, and let L,N respectively be the 
normal closures of S and S U {ae} in K. Noting that 

y'^aiy^ = a3a^\ y^a2y^ = \ y^asy^ = aio^S 

y^a^yS ^ 020605 ^a2^ 2/^052/^ = 0205 \ 2/^06?/^ = 020405^^02^^ 

we have that K/L ^ and ZC/iV ^ Z^, and hence that N/N' / 0. 

Let (j) : K ^ K he given by Oj i— > y'^aiy'^ (1 < i < 6). It is clear from the 
presentation of K that (/> is an automorphism of K\ furthermore 0(A^) = A^. In the 
abelian group K/N, <p{ai) = y'^aiy'^ = a^^ (1 < i < 6). That is, (f) induces the 
antipodal automorphism a —a on K/N. By Corollary 3.2 of K contains a 
non-abelian free subgroup. □ 

We will write the trace polynomial of T as t(A) = trti;(X, y), where tr{X) = 1, 
tr(y) = ^/2, X = tr(Xy). By Lemmas [l] and IS we may assume that trXy = or 
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X, Y generate ^4, ^4, or A^. But Y has order 4 so X, y cannot generate ^4 or A^. If 
X, Y generate 6*4 then the product XY has order 2 or 4 so tiXY = 0, ±V2. Suppose 
iiXY = —\f2. It follows from the identity 

trXy + trX^^y = (trX)(try) 

that trX^^y = 2\/2- Replacing X by X~^ in Lemma ^ shows that F contains a 
non-abelian free subgroup. Thus we may assume that the only roots A = trXy of r 
are A = 0, \/2. Using the leading coefficient of r is given by c = ±(1/2)'' where k 
denotes the number of values of % for which /3j = 2. Hence r(A) takes the form 

r(A) = - ^f-' (2) 

where s > 0. Moreover, Theorem 2 of [3] implies that the Main Theorem holds when 
k is even, so we may assume that k is odd. 
Let 

/ gi7r/3 \ / e^-^l^ Z \ 

\\ e— /3 J ' 1^ e— /4 ) ■ 

Then tr^ = 1, irB = ii AB = z - (^6 - \/2)/2. Consider the represen- 
tation p : { x,y \ x"^ = = 1 ) PSL{2, C) given hy x ^ A,y B. Then 
trp(x"^y^^ . . . x'^'^y^'") = t{z — — \/2)/2) whose constant term (by ©) is 

±{V2Y{{VQ - V2)/2y{{V^ + V2)/2f~' 

which simplifies to 

±(^/2)'^((^/6 + ^/2)/2)^-2^ 
Now the constant term in tr(^"ii?^^ . . . A°''=B^'=) is equal to 



2 cos 



12 



Thus {V2)^{{V6 + V2)/2))''-^' = 2cos (^iiE^ki^i+jE^iftK^ ^^^^^ ^ -g ^^^^ ^j^^g 
only happens if k = and A; — 2s = ±1. It follows that 

k k 

4^ai + 3^A = 1,5,7, 11 mod 12. (3) 

i=l i=l 

Since k = there is no value of i for which (3i = 2 and hence F maps homomor- 
phically onto the group 

f = ( X, y I = y2 = w{x, yf = 1) (4) 

where {^{x, y) = x°^^y . . . x°'''y. If ?D is a proper power then F contains a non-abelian 
free subgroup by [21. Thus we may assume that w is not a proper power, and so @ 
is a presentation of F as a generalized triangle group. 



4 



We will write the trace polynomial of T as cr{fi) = trid{X ,Y), where tr(X) = 1, 
tr(y) = 0, /i = tv{XY). It follows from that "^^^i Ui ^ mod 3 so f admits 
no essential cyclic representation. By Lemma H we may assume that fx = or X ,Y 
generate ^4,54, A5 or a finite dihedral group, in which case XY has order 2,3,4, or 
5 and hence = 0,±1,±V2, (±1 ± V5)/2. Moreover X is of order 4 in SL{2,C) 
so X-^ = -X and thus ti{X-^Y) = -/i and tiw{X,Y) = {-l)^tvw{X~^ ,Y), so 
f^wifJ-) = A*)- Thus /i and — /x occur as roots of a with equal multiplicity. 

By (P) the leading coefficient of a is ±1 so 

a{fi) = ±m"H/^' - - 2)"^(^' - (3 + V5)/2)"4(^2 _ (3 _ ^)/2)«5 

where ui, ^^3, ^^4, ^^5 > and ui + 2n2 + 2^3 + 2n4 + 2^5 = k. Since tr(Xy) is a 
polynomial with integer coefficients in trX = 1, try = we have that U5 = U4 so 

and ui + 2u2 + 2n3 + 4m4 = k. Let 




I z 
-i 



Then trA = 1, tri? = 0, ticAB = z — -v/3- Now the constant term in a{z — \/3) 
is (-\/3)"i • 2"2. But the constant term in tr(i"i^ . . . A'^'^B) is 2cos((2X;Li + 
3/c)7r/3) = ib\/3 so = I, n2 = and thus k = 1 -\- 2u^ + 4ti4. 



Lemma 3 // \/2 is a repeated root of cr{fj,) then F contains a non-abelian free sub- 
group. 

Proof 

Let g : r ^ r denote the canonical epimorphism. By hypothesis, there is an essential 
representation p -.T —>■ PS'L(2, C[/u]/(/u — \/2)^). Indeed, we can construct p explicitly 
via: 

Pi-) = ( ' ^0 e--/3 ) ' = ( ? V ) ■ 

Composing this with the canonical epimorphism 

ifj : PSL{2,C[p.]/{fi - V2f) PSL{2,C[f^]/{f^ - V2)) ^ PSL{2,C) 

gives an essential representation p = ip o p : T ^ PSL{2,C) with image 5*4, corre- 
sponding to the root \/2 of the trace polynomial. 

Let K denote the kernel of p, V the kernel of tp, and K the kernel of the composite 
map p o q : T ^ PSL{2,C). Then V is a complex vector space, since its elements 
have the form ±(1 + {p — \/2)A) for various 2x2 matrices A, with multiplication 

[±{I + {p- V2)A)][±{I + {p- V2)B)] = ±{I + {p- V2){A + B)). 



5 



Now K is generated by conjugates of (xy) 



1^ and p{{xy)^) 



/ + (/i - V2)M 




Since M is non-zero, K (and hence K) 



maps onto the free abehan group of rank 1. Let be a normal subgroup of K such 
that K/N ^ Z. 

Note that K is the fundamental group of a 2-dimensional CW-complex X arising 
from the given presentation of F. This complex X has 24 cells of dimension 0, 48 
cells of dimension 1, and 24(| + | + ^) = 26 cells of dimension 2. Here, 24/4 = 6 of 
the 2-cells (call them ai, . . . , ag) say) arise from the relator y'^, 24/3 = 8 (07, . . . , 014, 
say) arise from the relator x^, and 24/2 = 12 (ais, . . . , 026) say) arise from the relator 
w{x,y)'^. Moreover, ai, . . . ,05 are attached by maps which are 2nd powers. Let X 
be the regular covering complex of X corresponding to the normal subgroup N K 
and let Sjj denote a lift of the 2-cell a^. Then each of Si, . . . , is a 2-cell attached 
by a map which is a 2nd power. 

Let GF2 denote the field of 2 elements. Now H2{X,GF2) is a subgroup of the 
2-chain group C2{X, GF2) and since K/N freely permutes the cells of X, C2{X, GF2) 
is a free GF2(-fC/A^)-module on the basis Si, . . . ,S26- Let Q be the free GF2{K/N)- 
submodule of C2{X ,GF2) of rank 6 generated by Si,...,S6. Since these 2-cells 
are attached by maps which are 2nd powers, their boundaries in the 1-chain group 
Ci{X ,GF2) are zero. Thus Q is a subgroup of H2{X ,GF2). Since the rank of Q is 
greater than x(^) = 2, Theorem A of IT implies that K, and hence F, contains a 
non-abelian free subgroup □ 

Lemma 4 If{l + VE)/2 is a repeated root of a{fi) then F contains a non-abelian free 



The proof is similar to that of LemmaOl In this case p has image A^, corresponding to 
the root {l + V^)/2. The complex X has 60 0-cells, 120 1-cells, and 60{\ + l + ^) = 65 
2-cells (so xi^) = 5). Moreover, 60/4 = 15 of the 2-cells (call them ai,...,ai5, 
say) are attached by maps which are 2nd powers. As before, the free GF2{K /N)- 



implies that K contains a non-abelian free subgroup. □ 

By Lemmas 01 and 13 we may assume ^3, M4 < 1 so < 7. A computer search 
reveals that if fe = 3 or 7 then there is no word w{x, y) such that t(A) is of the 
form ((21). If A; = 5 then (up to cyclic permutation, inversion, and automorphisms 
of ( X I ) and ( y | )) the only word w{x,y) with t(A) of the form is 
w = xyxyx^y^x^yxy^ . In this case, a computer search using GAP |9j shows that F 
contains a subgroup of index 4 which maps onto the free group of rank 2. If A; = 1 
then either F = ( x, y | x^ = = (xy)^ = 1 ) or F = ( x, y | x^ = y^ = (xy^)^ = 1 )• 
In the first case F = 5*4, and in the second F can be written as an amalgamated free 



subgroup. 



Proof 



submodule, Q, of C2(X,GF2) of rank 15 generated by Si, . . . ,Si5 is a subgroup of 
H2{X ,GF2). Since the rank of Q is greater than xi^)i Theorem A of again 
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product 

r = ( x,y2 I ^3 = / = {xy^f = 1 ) * ( y I / ) 

in which the amalgamated subgroup has index 3 in the first factor and index 2 in the 
second, and thus F contains a non-abelian free subgroup. This completes the proof 
of the Main Theorem. 
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